Distributed linear regression by averaging

Distributed statistical learning problems arise commonly when dealing with large datasets. In this setup, datasets are partitioned over machines, which compute locally, and communicate short messages. Communication is often the bottleneck. paper, we study one-step iterative weighted parameter averaging in linear models under data parallelism. We do regression on each machine, send results to a central server take average of parameters. Optionally, iterate, sending back doing local ridge regressions centered at it. How does work compared full data? Here, performance loss estimation test error, confidence interval length high dimensions, where number parameters comparable training size. find averaging, also give for averaging. that different affected differently by distributed framework. Estimation error increases lot, while prediction much less. rely recent from random matrix theory, develop new calculus deterministic equivalents as tool broader interest.